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EXECUTIVE  SUMMARY 


This  report  documents  the  analytical  results  and  conclusions  of  a  four-month 
investigation  of  cost  and  uncertainty  in  selected  DARPA  programs.  The  analysis 
focuses  on  risk  and  the  management  of  risk  from  a  technical,  cost,  and  schedule 
perspective  through  a  comparison  of  DARPA  experience  to  a  large  experience  base 
of  other  federal  programs.  The  results  Indicate  that  DARPA  experience  in  the 
management  of  high  technology  programs  can  be  interpreted  (and  to  some  degree 
of  accuracy,  forecasted)  using  this  experience  base.  In  addition,  this 
analytical  and  empirical  approach  can  be  used  to  assist  DARPA  in  complying  with 
recent  DGD  directives  to  recognize  risk  in  budgeting  and  planning  estimates. 

Most  importantly,  the  study  effort  demonstrates  that  it  is  possible  to  Identify 
qualitative  and  quantitative  indicators  to  characterize  risk  and  to  use  such 
descriptors  as  the  basis  for  risk  management. 

/Tx 

The  analysis  described  in  this  report  ib  based  in  large  part  on  a  data  base 
related  to  federal  historical  experience  in  hundreds  of  R&D  programs  over  a 
number  of  years.  Usiug  this  data  in  concert  with  information  related  to  the 
budgetary  history  of  selected  DARPA  programs,  a  technique  to  estimate  risk 
budgets  as  a  function  of  confidence  level  was  developed.  Finally,  using  this 
technique  as  a  point  of  departure,  a  management  process  to  (1)  integrate  manage¬ 
ment  input  to  modify  and  revise  risk  budget  estimates  and  (2)  coordinate  existing 
coBt/schedule/performance  program  data  into  the  control  of  risk  budgets  was 
designed.  The  process  is  designed  to  be  simple,  iterative,  and  amenable  to 
subjective  input  by  management.  It  was  also  developed  so  as  to  require  minimal 
resources  to  Implement  and  to  fully  utilize  existing  DARPA  management  informa¬ 
tion  systems  upon  which  current  programming  and  budgeting  activities  are  based. 
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I .  INTRODUCTION 


A.  SCOPE 

This  report  documents  the  results  of  technical  efforts  to  analyze  the  size 
and  natute  of  cost  growth  and  cost  growth  risk  in  selected  DARPA  R&D  programs 
and  to  develop  a  recommendation  for  monitoring  and  controlling  such  risk.  An 
Interim  Progress  Report  previously  submitted  discussed  the  results  of  a 
statistical  comparison  of  historical  data  related  to  a  wide  range  of  government 
R&D  programs,  including  DARPA  activities.  That  effort  identified  the  develop¬ 
ment  of  a  procedure  whereby  cost  growth  risk,  expressed  as  a  level  of  confidence 
in  the  adequacy  of  a  program  risk  budget  of  a  given  size,  could  be  quantitatively 
estimated  for  a  set  of  R&D  programs.  This  report  expands  on  this  concept  and 
describes  a  management  framework  within  which  such  procedures  can  be  used  to 
continually  assess  and  compare  cost  growth  risk  as  programs  evolve  in  an  attempt 
to  improve  DARFA  capability  to  anticipate,  understand,  and  mitigate  cue  impact 
of  uncertainty  in  programs  characterized  bv  risk. 

B.  OBJECTIVE 

This  effort  was  conducted  under  the  auspices  of  the  DARPA  Program  Management 
Office.  It  is  the  responsibility  of  this  office  to  plan,  manage,  and  control, 
at  the  aggregate  level,  the  allocation  of  DARPA  program  funds.  This  management 
responsibility  is  separate  and  distinct  from  the  responsibilities  of  the  technical 
program  offices  to  manage  projects  from  a  technical,  cost,  and  schedule  point 
of  view.  Given  the  overall  DARPA  mission  to  pursue  high  risk,  high  payoff  R&D, 
it  is  Incumbent  upon  top  management,  including  the  Office  of  the  Director  and 
the  Program  Management  Offle,  to  plan  and  be  prepared  for  the  possible  occurrence 
of  significant  cost  growth  in  on-going  DARPA  piograms.  The  most  effective  way 
to  satisfy  this  requirement  is  through  the  establishment  and  management  of  a 
program  risk  budget  control!  A  by  the  Director. 


1-1 


The  objective  of  the  effort  documented  in  this  report  is  to  develop  =»  set  of 
analytical  techniques  which  can  be  used  as  the  basis  to  identify  a  management 
structure  and  set  of  procedures  to  be  implemented  by  the  Program  Management 
office  through  which  such  a  budget  can  be  established  and  controlled.  This 
management  process  has  been  designed  so  that  minimal  additional  requirements 
for  data  and  reporting  would  be  placed  on  the  technical  program  offices.  In 
addition,  it  is  anticipated  that  implementation  within  the  Program  Management 
Office  could  be  achieved  using  information  available  from  management  systems 
currently  in  place  or  under  development  and  that  operation  would  place  little 
or  r.o  additional  workload  on  the  limited  staff  resources  of  the  Office.  The 
overall  process,  described  in  detail  in  Section  II,  is  an  iterative,  interactive 
one  in  which  an  initial  aggregate  level  of  funding  for  risk  is  established 
and  subsequently  adjusted  and  allocated  among  selected  programs  based  on  a 
systematic  process  to  assess  relative  cost  and  schedule  risk  within  them.  The 
remainder  of  this  section  will  provide  a  brief  description  of  risk  and  cost 
growth  in  government  BUD  programs  and  relate  this  environment  of  uncertainty  to 
th<a  context  of  the  DARPA  mission. 

C.  BACKGROUND 
1.  General 

The  pervasiveness  and  magnitude  of  uncertainty  and  risk  in  government 
R&D  programs  and  its  Impact  o.i  cost  overruns  and  schedule  slippages  is  widely 
recognized.  Conceptually,  thene  relationships  can  be  depicted  as  in  Figure  1-1. 
Although  the  interaction  among  technical  performance,  cost,  and  schedule  may 
appear  intuitively  obvious,  quantification  of  such  relationships  to  the  point 
that  accurate  predictions  for  a  specific  R&D  program  can  be  performed  is  a 
difficult  task.  Design  of  specific  models  is  necessary  to  develop  the  cost- 
performrnce  curves.  Likewise,  networks  Identifying  critical  activities  and 
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their  associated  durations,  prerequisites  and  costs  are  required  for  cost-schedule 
and  performance-schedule  trade-offs.  Within  DARPA,  as  development  of  technical 
programs  proceeds,  additional  Information,  Including  design  models  and  networks 
can  become  available  to  allow  a  better  understanding  of  the  shape  of  the  curves 
and  the  trade-offs  among  variables.  As  risk  analyses  continue,  more  information 
on  which  to  postulate  the  width  of  the  bands  can  also  be  developed.  However, 
at  the  Inception  and  early  phases  of  such  programs  (at  a  time  when  initial 
cost  estimates  are  made  and  funds  committed),  the  data  necessary  to  construct 
these  curves  or  the  bands  of  uncertainty  around  them  for  any  program  are 
typically  not  available. 

Nonetheless,  it  is  possible  to  discuss  the  impact  of  schedule  and 
performance  variations  in  terms  of  implications  for  project  cost.  Using  the 
potential  for  cost  overrun  as  an  indicator  of  risk  addresses  a  major  concern 
of  DARPA  management  -  the  ultimate  cost  to  completion.  Significant  cost  overruns, 
while  not  unusual  for  recent  Federal  R&D  programs,  could  prove  critical;  and 
unacceptably  high  overruns  could  lead  to  the  cancellation  of  a  project.  This 
coacern  for  anticipating  and  preventing  cost  overruns  is  validated  by  the 
experiences  of  other  technology  development  programs. 

There  is  significant  empirical  data  to  document  the  extent  of  cost 
overruns  in  Federal  R&D  programs.  Statistical  analyses  of  the  extent  of  cost 
growth  in  a  large  number  of  programs  were  presented  in  the  Interim  Progress 
Report.  The  delta  used  for  this  quantification  came  from  a  comprehensive  data 
base  developed  by  the  Meridian  Corporation.  Portions  of  this  data  are  recorded 
in  Appendix  A,  summarizing  combined  military  and  civil  major  acquisitions  fcr 
the  primary  purpose  of  disclosing  the  financial  status  of  major  Federal  pregrams. 
Included  in  this  data  are  acquisitions  either  in  the  development,  test,  production 
or  construction  phases.  Major  programs  as  defined  by  the  Department  of  Difense 


include  those  in  excess  of  $75  million;  civil  projects  are  those  in  excess  of 
$25  million. 

Figure  1-2  shows  the  cost  growth  it.  civil  and  military  acquisitions 
since  1975.  Persistent  cost  growth  is  reflected  in  the  shaded  areas  which 
depict  the  differences  between  the  current  cost  estimate  and  the  planning 
estimates  for  the  projects  included  in  each  year's  data.  The  original  estimate 
was  defined  as  the  estimate  used  for  program  planning.  The  baseline  estimate 
is  the  estimate  ac  the  beginning  of  development.  The  current  estimate  is  the 
most  recent  estimate  available  in  each  year.  In  Figure  1-2,  ”n”  represents 
the  total  number  of  major  acquisitions  for  that  year.  For  example,  in  1980 
there  were  1040  total  projects  examined.  Significant  cost  growth  occurred  in 
131  of  those  projects.  The  reasons  for  overruns  in  these  projects  are  further 
discussed  below  and  depicted  in  Figure  1-3. 

The  collected  data  have  been  analyzed  in  terms  of  causes  of  cost 
growth  for  civil  acquisitions  having  100  percent  or  more  increases  and  selected 
military  acquisitions.  The  cost  increases  are  attributed  to  the  following 
seven  factors: 

1.  Quantity  -  changes  including  scope. 

2.  Engineering  -  changes  altering  a  system's  established  physical  or 
functional  characteristics. 

3.  Support  -  changes  involving  spare  parts,  ancillary  equipment, 
warranty  provisions,  and  government-furnished  property  or  equipment. 

4.  Schedule  -  changes  in  delivery  schedules,  completion  date  or  some 
intermediate  milestone  of  development,  production  or  construction. 

5.  Economic  -  changes  that  are  influenced  by  one  or  more  factors  in 
the  economy,  such  as  inflation. 

6.  Estimating  -  corrections  or  other  changes  occurring  since  the  initial 
or  other  baseline  estimates  for  program  or  project  cos's 

7.  Sundry  -  changes  other  than  the  above  categories,  such  as  environmental 
costs  and  relocation  assistance  for  water  and  highway  projects. 


1-5 


FIGURE  1-2 
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Figure  1-3  illustrates  the  relative  impact  of  these  seven  GAO  causal 
factors  on  cost  overruns  in  Federal  major  systems  acquisitions. 

Taken  together,  these  two  figures  illustrate  several  important  points 
of  interest  to  DARPA  management.  First,  the  evidence  over  time  points  to 
a  growing  inability  government-wide  to  accurately  predict  or  avoid  significant 
cost  overruns.  This  trend  persists  for  all  classes  of  high  technology  programs 
in  defense,  energy,  and  space  applications.  Analyses  of  these  overruns  of  the 
type  generally  depicted  in  Figure  1-3  indicate  that  all  three  of  the  ke,  variables 
shown  in  Figure  1-1  —  cost,  schedule,  and  technical  performance  —  are  highly 
inter-related.  However,  the  complexity  of  these  relationships  has  prevented 
the  isolation  of  causal  factors  and  quantification  of  simple  analytical  predictors. 
Neverthelesc ,  due  to  the  large  available  data  base,  the  concept  of  cost  growth 
risk  provides  the  best  overall  indicator  of  program  risk  for  management.  This 
concept  and  its  detailed  quantification  for  DARPA  were  fully  described  in  the 
Interim  Progress  Report.  Its  application  to  the  task  of  developing  an  on-going 
risk  management  process  for  DARPA  management  is  described  in  Section  II, 

2.  Implications  for  POD  and  DARPA 

The  general  problem  of  ‘isk  assessment  is  especially  germane  to  DARPA 
management  at  this  time.  In  April,  1981  the  Deputy  Secretary  of  Defense 
promulgated  a  set  of  thirty-one  recommendations  to  modify  and  improve  the 
efficiency  of  the  defense  acquisition  management  and  procurement  process.* 

Among  these  recommendations  were  several  that  relate  to  risk  management  and 

assessment.  Recommendation  No.  6  suggests  that  the  Services  be  required 

"to  budget  to  most  likely  or  expected  costs,  including  predictable  cost  increases 


♦Department  of  Defense  Memorandum,  April  30,  1981. 
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due  to  risk."  Recommendation  No.  11  provides  that  there  be  ”Increase[d]  DoD 
efforts  to  quantify  risk  and  expand  the  use  of  budgeted  funds  to  deal  with 
the  uncertainty.”  Beth  of  these  recommendations  are  consistent  with  DARPA' s 
role  as  the  DoD  agent  responsible  for  investigating  the  potential  of  long 
term,  Innovative,  technologically  advanced  concepts  to  nationa]  defense 
requirements.  As  such,  DARPA  sponsors  programs  to  proof  of  concept  which 
are  inherently  subject  to  a  high  degree  of  technical  risk  and  uncertainty. 

In  particular,  in  recent  years,  DARPA  has  been  sponsoring  programs  which 
involve  prototype  hardware  systems  development  and  experimentation.  These 
programs  [e.g.,  the  DARPA  Program  Element  entitled  Experimental  Evaluation 
of  Major  Innovative  Technologies  (EEMIT)]  tend  to  involve  relatively  large 
annual  budgets  and  are  subject  to  a  great  deal  of  uncertainty  in  cost  estimates. 
As  such,  they  are  primary  candidates  for  application  of  an  on-going  risk 
assessment  and  management  process  within  the  agency. 
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II.  MANAGEMENT  PROCESS  DEVELOPMENT 


A.  DARPA  REQUIREMENT 

Explicit  OSD  direction  and  DARPA' s  growing  involve  int  in  high  risk,  high 
visibility,  hardware  development  programs  virtually  dictate  a  requirement  for 
a  system  of  risk  assessment  and  management  applicable  to  the  EEMIT  programs. 
Following  both  logic  and  OSD  direction,  the  DARPA  goal,  as  a  first  step, 
should  be  the  establishment  of  a  "risk  budget”  for  the  EEMIT  programs.  The 
DARPA  Program  Management  Office  (PMO)  currently  mantains  a  system  of  cost 
growth  monitoring  that  can  serve  as  the  administrative  and  philosophical  founda¬ 
tion  for  a  justified  and  flexible  risk  budget.  A  systematic  program  of  risk 
assessment  and  risk  management  could  result  in  the  establishment  of  relative 
levels  of  program  risk,  or  alternatively  levels  of  confidence  of  cost  overrun 
avoidance.  These  risk,  or  confidence  levels  may  then  be  used  to  establish  a 
justifiable  allocation  of  funds  into  the  desired  risk  budget. 

While  using  the  philosophy  of  "budgeting  for  risk,"  there  is  a  requirement 
to  clarify  the  relationship  between  overall  program  risks  and  budgeted  funds. 
Generally,  program  risks  are  reduced  to  three  broad  categories  -  cost,  schedule 
and  technical  (oc  performance)  risk.  Technical  risk  Is  difficult  to  measure 
prior  to  certain  program  milestones  and  testing  of  the  "product",  and 
then  performance  measurements  are  usually  program-unique  and  may  not  be  directly 
compared  with  dissimilar  technologies.  Similarly,  schedule  risk  is  intimately 
related  to  both  cost  and  technical  risk,  and  quite  difficult  to  measure  or 
predict  independently  prior  to  the  occurrence  of  an  actual  slippage.  It  is 
also  recognized  that  cost  risk,  schedule  risk  and  technical  risk  are  to  some 
degree  interdependent  in  all  programs,  but  that  due  to  statistical  uncertainty 
the  interrelationship  is  rarely  quantifiable  or  functionally  specific.  The 
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accepted  "common  denominator"  measure  of  program  efficiency  is  cost*  One  may 
rationally  argue  that,  in  general,  realized  schedule  and  technical  risks  will 
ultimately  generate  cost  increases  or  overruns  of  predicted  budget  level 
requirements . 

Cos t  risk,  or  more  specifically,  "cost-growth"  risk  may  therefore  be 
considered  a  fairly  representative  and  accurate  indicator  of  overall  program 
risk,  directly  relatable  to  a  "risk  budget."  This  does  not  imply  that  poten¬ 
tial  variances  in  schedule  and  performance  objectives  should  nor  be  monitored 
as  lead  indicators  of  cost  growth. 

Having  determined  that  cost-growth  risk  may  be  used  as  an  operative  measure 
of  relative  program  risk  or  confidence  to  be  used  initially  in  establishing  a 
risk  budget,  a  quantifiable  measure  of  risk  and/or  confidence  level  is  required. 
Additionally,  practical  experience  and  the  historical  record  demonstrate  that 
program  risk  is  a  dynamic  variable  and  must  be  monitored  and  reassessed  in  an 
interative  manner  throughout  the  duration  of  each  program  if  the  PMO  is  to 
have  the  capability  to  manage  the  risk  and  balance  the  risk  budget  between 
programs.  There  is  therefore  a  need  for  an  iterative  and  program  interactive 
process  of  risk  assessment  and  management. 

U.  RELATIVE  MEASURE  OF  RISK/CONFIDENCE 

The  rajor  preliminary  requirement  for  the  establishment  of  the  desired 
risk  assessment  and  management  process  is  a  definitive  measure  of  .isk  or 
confidence  level.  A  "cost-growth  risk”  relationship  has  been  developed  which 
provides  a  relative,  but  quantitative,  measure  of  risk  based  on  the  historical 
experience  of  major  high  technology  programs  in  DOD,  DOE,  NASA  and  DARPA.  Tnis 
relationship  was  established  by  detailed  analysis  of  available  government-wide 
(including  DARPA)  data  on  program  cost  estimates,  their  iterative  growth  and 
actual  realized  costs. 


II-2 


Initial  analysis  of  the  data  base  resulted  in  the  determination  that  the 


relevant  measure  of  cost-risk  was  not,  in  fact,  the  risk  of  the  occurrence  of 
a  cost  overrun,  since  virtually  all  programs  in  the  data  base  experienced 
repetitive  yearly  cost  overruns.  The  relevant  measure  of  program  risk  is 
rather  the  risk  of  additional  "cost-growth"  once  a  cost  overrun  has  been 
predicted,  or  the  occurrence  of  significant  ovarruns  in  any  given  program 
assumed.  This  is  in  line  with  the  basic  OSD  direction,  that  assumes  some 
overruns  are  inevitable  dus  to  the  inherent  "risk"  in  high  technology  programs, 
and  that  "risk"  must  be  quantified  and  budgeted  for,  l.e.  determine  the  total 
potential  amount  of  overrun,  budget  for  it,  and  control  it  as  it  occurs. 

The  overrun  data  in  the  DOD-DOE-NASA  portion  of  the  data  base  was  treated 
independently  of  the  DARPA  data  to  determine  the  consistency  of  the  smaller 
DARPA  base  with  the  larger  and  more  statistically  valid  government-wide  data 
base . 

The  data  base  was  first  subjected  to  "delphic"  analysis  to  segregate  "high," 
"medium"  and  "low”  risk  programs,  based  on  technology  application.  High  and 
medium  risk  programs  for  FY  '77,  '78,  '79  and  '80  were  examined  in  detail. 

Overrun  magnitude  was  represented  in  dimensionless  "cost  factor"  form,  that 
is  the  ratio  of  the  year-end  estimate,  or  actual  cost,  to  the  original  program 
baseline  cost  prediction  for  that  year.  After  eliminating  obviously  discrepant 
data  points  such  as  those  known  to  relate  to  program  cancellations,  over  six 
hundred  usable  cost  factors  were  obtained. 

An  analysis  was  conducted  to  determine  the  natural  aggregations,  or 
groupings  of  the  data  points  to  determine  if  there  were  significant  causal 
factors  of  overruns  evident  in  the  data  base.  The  results  of  this  effort  were 
generally  negative,  as  there  were  no  obvious  groupings  of  data  by  program 
type  (aircraft,  ship,  satellite,  etc.)  or  technology  level,  or  even  agency. 
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However,  displaying  the  data  by  ’spectral  density"  (in  l.  one-dimensional  plot 
of  cos t  factor  magnitude)  for  each  fiscal  year,  resulted  in  a  consistent 
progression  of  aggregations  of  closely  bunched  data  points.  Using  the 
statistical  relationships  for  the  "normal  distribution,”  the  mean  value, 
and  standard  deviation  were  calculated  for  each  data  aggregation.  This 
resulted  in  ten  statistical  data  points.  Since  the  value  for  twice  the 
standard  deviation  is  representative  of  a  orobability  of  non-occurrence  of 
97. 5%,  this  value  was  considered  representative  of  a  confidence  level  of 
non-occurrence  of  cost-growth  of  approximately  100%,  when  added  to  the  mean 
value.  Simply  stated,  for  a  cost  overrun  predicted  to  be  at  the  mean  value, 
one  could  be  97.5%  certain  that  the  actual  overrun  would  exceed  the  mean 
value  plus  twice  the  standard  deviation,  based  on  statistical  analysis  of  the 
historical  data  base.  •  This  relationship  is  plotted  in  Figure  11-1  as  "DOD-DOE- 
NASA"  data  represented  by  circular  data  points.  Eight  additional  statistical 
data  points  were  developed  by  this  process  from  the  DOD-DOE-NASA  data  base  from 
those  programs  where  the  components  of  the  cost  overruns  were  listed  in  the 
data  baes.  In  this  case  "quantity"  and  "production"  components  of  the  overruns 
were  deleted  in  calculating  the  cost  factors,  resulting  in  data  more  representa¬ 
tive  of  overruns  Incurred  in  developmental  programs.  These  data  are  represented 
as  squares  in  Figure  II-l  labeled  "DOD-DOE-NASA  developmental."  These  two  sets 
of  data  resulted  in  a  definitive  and  credible  cost-growth  risk  trend  represented 
by  the  lower  dashed  lines  in  Figure  II-l. 

Next  the  available  DARPA  EEMIT  data  base  was  processed  in  the  same  >..anner 
as  the  previous  sets,  with  the  exception  that  the  cost  factors  were  formed 
exclusively  of  the  ratio  of  actual  overruns  to  predicted  overruns.  The  DARPA 
statistical  aggregations  are  represented  by  "X"’s  in  Figure  II-l,  and  are  highly 
consistent  with  the  DOD-DOE-NASA  data  and  experience,  with  the  bias  toward 
developmental  programs,  as  would  be  anticipated.  The  trend  displayed,  and  the 
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FIGURE  II-l 
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selection  of  a  "cost-growth  risk"  relationship  as  the  relative  measure  of  risk 
or  confidence  level  appear  to  be  totally  applicable  to  DARPA  EEMIT  program 
experience.  For  the  DARPA  EEMIT  data  the  uppermost  dashed  line  of  Figure  11-1 
is  representative  of  the  "100%  confidence  level  of  cost  growth  avoidance"  or 
alternatively,  "zero  risk  of  cost  growth."  Graphically,  this  can  be  interpreted 
as  follows  (in  Figure  II-l)  given  a  predicted  cost  increase  of  say,  50%,  result¬ 
ing  in  a  cost  factor  of  1.5,  selecting  1.5  on  the  horizontal  axis,  proceeding 
vertically  to  the  "DARPA”  line,  and  then  horizontally  to  the  vertical  axis  and 
a  value  of  approximately  2.5,  signifying  that  with  a  predicted  cost  increase 
of  50%,  statistical  cost-growth  experience  indicates  that  "risk  budget  funds" 
of  150Z  would  be  required  to  provide  100%  confidence  of  not  realizing  an 
actual  cost  overrun.  Application  of  this  relationship  to  the  formulations  for 
the  "normal  distribution"  provides  the  "cost-growth  risk"  relationship  whereby 
entering  a  risk  fund-  amount  (in  percentage  or  decimal  value)  the  level  of 
either  "confidence  of  cost  growth  avoidance"  or  "cost  growth  risk"  will  be 
obtained  (again  in  dimensionless  form) . 

The  cost-growth  risk  relationship  will  be  used  throughout  the  proposed 
risk  assessment  and  management  process  as  the  "relative  measure  of  risk" 
whereby  risk  and/or  confidence  levels  will  be  expressed,  providing  the 
unifying  measure  of  effectiveness  for  the  analysis. 

C.  INITIAL  RISK  BUDGET 

The  establishment  of  an  initial  "justiried"  risk  budget  for  EEMIT  programs 
is  a  necessary  prerequisite  to  the  proposed  process  of  risk  assessment  and 
risk  management.  Initial  levels  of  confidence  must  exist  as  relative  measures, 
or  targets,  which  the  risk  analysis  and  management  phases  will  ultimately 
refine  and  develop  into  firmly  justified  risk  budget  allocations.  The  initial 
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risk  budget  (or  risk  fund  allocation)  will  be  established  by  the  Program  Manage¬ 
ment  Office  (PMO)  and  the  Director  for  the  POM. 

Distribution  factors  based  on  program  funding  profiles  are  then  established. 
Each  program  is  allocated  a  distribution  facto1*  equal  to  the  decimal  value  of 
that  program’s  fractional  share  of  the  total  EEMIT  budget.  The  total  EEMIT 
risk  fund  is  then  multiplied  by  each  distribution  factor  (decimal  value)  produc¬ 
ing  a  risk  budget  for  each  program  based  solely  on  budget  share. 

Similarly,  multiplying  ee^h  program's  POM  year  budget  values  by  this 
distribution  factor  will  result  in  the  program  risk  budget  distribution  across 
the  POM,  again  based  solely  on  budget  share  and  distribution  per  year. 

Since  it  is  recognized  that  many  factors  other  than  budget  share  will  be 
influential  ir,  determining  the  desired  confidence  level  (risk  fund  allocation) 
for  a  program,  influence  factors  based  on  non-specific  program  characteristics 
will  be  determined  by  consensus  review  of  competing  EEMIT  programs  by  the 
Directors  of  the  six  DARPA  "Technology"  Offices  or  their  designated  representa¬ 
tives.  The  six  Directors  or  their  representatives  will  be  afforded  the  oppor¬ 
tunity  to  review  the  EEMIT  programs  during  a  dedicated  series  of  briefings  by 
the  program  managers. 

After  the  briefings  the  Directors  will  be  allowed  two  weeks  in  which  to 
evaluate  all  the  EEMIT  programs  on  evaluation  sheets  issued  by  the  PMO. 

The  evaluation  sheets  will  define  rating  categories  for  the  programs  such 
as  —  political  impact  —  operational  potential  —  technical  uniqueness  — 
existing  requirement  —  program  longevity  —  ’’personal  preference"  —  and 
any  other  categories  desired  or  preferred  by  the  PMO.  Each  rating  category 
will  be  assigned  a  numerical  decimal  value  between  zero  and  one  by  each  Director 
for  each  EEMIT  program.  The  values  for  each  program  will  be  added  and 
non-dlmen8lonalized  into  an  overall  decimal  value  on  each  rating  sheet,  and 
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all  the  sheets  for  each  program  in  turn  averaged  Into  a  representative  decimal 
value  for  that  program. 

With  this  "influence"  factor  established  for  each  program,  it  will  then 
be  multiplied  with  the  previously  determined  funding  profile  distribution 
factor  for  that  same  program. 

The  resulting  combined  factors  are  then  non-dimensionalized  by  dividing 
each  of  them  by  the  sum  of  all  the  combined  factors  resulting  in  a  set  of 
"adjustment"  factors  for  risk  fund  distribution,  each  of  which  has  a  decimal 
value  less  than  one,  and  the  sum  of  which  is  equal  to  one. 

The  adjustment  factor  for  each  program  will  then  be  multiplied  by  the 
overall  EEMIT  risk  fund  to  obtain  that  program's  risk  budget,  and  by  each 
program's  yearly  POM  budget  to  obtain  the  yearly  allocation  of  risk  budget 
within  the  program,  now  based  both  on  "distribution"  and  "influence”  factors. 

The  resulting  allocation  of  the  overall  risk  fund  is  the  "initial  risk 
budget,"  justified  both  by  funding  profile  and  subjective  judgment. 

D.  THE  RISK  ASSESSMENT  AND  MANAGEMENT  PROCESS 

Once  the  initial  risk  budget  has  been  established  for  all  (or  selected) 

EEMIT  programs,  a  continuing  flow  of  information  will  occur  from  the  DARPA 
Program  Offices  to  the  Program  Management  Office  then  on  to  the  Director  and 
finally  back  to  the  Program  Offices.  The  following  is  a  step-by-step  description 
of  the  program  illustrated  in  Figure  II-2. 

The  Program  Offices  will  provide  the  Program  Management  Office  with: 

•  program  funding  profiles 

•  program  work  breakdown  structures  (WBS) 

•  program  tim^-llne  milestone  networks 

•  assessments  of  percentage  cost  and  schedule  risk  for  each  WBS  element 
at  level  3  (this  risk  assessment  will  originate  at  the  management 
level  most  directly  responsible  for  each  WBS  element  assessed) 
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FIGURE  II-2 
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•  timely  prediction  of  the  extent  of  any  anticipated  cost  or  schedule 
"growth"  associated  with  each  WBS  level  3  element 

•  extent  of  any  cost  or  schedule  growth  in  each  UBS  level  3  element 
when  it /they  occur 

The  Program  Management  Office  will  have  at  hand: 

•  an  established  and  justified  risk  budgoc  based  on  the  most  recent 
POM  submission  for  ths=  EEMIT  programs 

•  an  established  "cost-growth  risk"  relationship  based  on  the  DOD-DOE- 
NASA  cost  overruu  experience  as  it  relaces  to  the  DARPA  cost  overrun 
history 

•  established  levels  of  "confidence  of  cost  overrun  a. jidance" 
(confidence  levels)  for  each  EEMIT  program  each  POM  year,  derived 
by  applying  the  cost-growth  risk  relationship  to  the  risk  budget 

The  Risk  Assessment/Management  Program  will  then  be  conducted  by/for  the 
Program  Management  Office. 

The  program  time-line  milestone  network  will  be  combined  with  WBS  level-3 
elements  into  a  "modified  critical  path"  (MCP)  network,  wherein  each  element 
has  an  associated  level  of  cost,  and  schedule  risk  provided  by  the  Program 
Office . 

The  initial  inherent  program  risk  level  will  then  be  established  by  mathe¬ 
matically  combining  and/or  compo  indlng  the  initial  WBS  risk  levels  along  the 
MCP  network,  as  well  as  "escalating"  or  "deflating"  any  WBS  estimates  that  are 
obviously  inconsistent  with  the  rest  'i  the  program  or  past  experience,  using 
the  cost-growth  risk  relationship. 

The  initial  program  risk  level  will  then  be  compared  with  the  established 
budgetary  confidence  level  for  each  program  year. 

This  process,  when  repeated  for  all  EEMIT  programs  will  give  the  flrft 
indications  of  the  adequacy  and  balance  of  both  the  initial  and  subsequent 
risk  budgets. 

When  cost  overruns  are  predicted  they  will  he  projected  into  the  MCP  and 
their  potential  impact  on  program  confidence  levels  established,  both  at  ;he 


level  of  overrun  predicted  and  at  an  escalated  "worst  case"  level  established 
from  the  cost-growth  risk  relationship. 

The  same  process  would,  of  course  be  repeated  for  actual  cost  overruns  at 
the  WBS  level  3  in  the  MCP  and  the  decrease  in  relative  program  confidence 
level  (established  by  the  allocated  risk  budget)  determined. 

Similarly,  both  predicted  and  realized  schedule  overruns  (extensions)  would 
be  projected  into  the  MCP  network  in  the  same  manner  as  cost,  but  without 
"worst-case"  escalated  values,  and  resulting  in  an  assessment  of  risk  of  meeting 
program  time  milestones. 

When  either  projected  or  actual  cost  or  schedule  overruns  threaten  to  in¬ 
crease  overall  program  risk  beyond  budgeted  levels,  the  MCP  and  WBS  will  be 
analyzed  for  potential  risk  reduction  and/or  risk  avoidance  options  such  as 
alternate  time-line  paths  or  WBS  element  phasing. 

Risk  reduction  options  so  determined  will  then  be  recommended  by  the 
Program  Management  Office  directly  to  the  program  manager,  or  indirectly  through 
the  Director,  and  the  six  Technology  Offices. 

When  projected  program  cost  risks  threaten  to  exceed  the  budgeted  confidence 
levels,  the  projected  risk  for  all  programs  must  be  examined  ani  the  risk 
budget  allocations  readjusted  or  increased  t:o  retain  desired  pr  ,ram  confidence 
levels . 

While  the  described  risk  analysis  and  management  process  is  essentially 
continuous  and  iterative,  a  "formal"  exercising  of  the  system  is  recommended 
at  least  bi-annually,  timed  to  the  POM  cycle. 

This  process  can  potentially  provide  responsive  Iterations  of  the  risk 
budget  whenever  required  as  well  as  interactive  risk  avoidance  and  reduction 
initiatives  between  the  Program  Offices  and  the  Program  Management  Office,  and 
"real  time"  projections  of  overall  program  risk  levels. 
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III.  SUMMARY  AND  CONCLUSIONS 


A.  THE  DARPA  REQUIREMENT 

By  its  very  nature,  DARPA  has  an  inherent  need  for  a  system  of  risk 
assessment  and  management.  DARPA  exists  primarily  to  manage  those  high  risk, 
high  potential  pay-off  programs  that  could  not  survive  budgetary  scrutiny 
in  the  research  and  technology  elements  of  the  individual  services.  Over  the 
years  DARPA  has  been  highly  successful  in  the  harvesting  of  high  pay-off 
technologies  while  weathering  the  attendant  high  risk.  This  relative  success 
has  resulted  in  DARPA  programs  growing  from  the  arena  of  generalized  technology 
exploitation  into  the  multiple  technology,  prototype-oriented  and  highly 
visible  world  of  present  EEMIT  programs.  OSD's  explicit  direction  to  identify, 
budget  for,  and  minimize  risk  is  particularly  appropriate  to  DARPA,  since  over 
the  years  that  agency  has,  in  effect,  been  the  DOD  specialist  in  the  management 
of  high  risk  programs.  The  analysis  conducted  of  the  historical  cost  overrun 
data  base  indicates  that  DARPA’ s  experience  has  been  statistically  comparable 
to  other  agencies  dealing  with  high  technology  development.  It  follows  that 
DARPA  should  be  the  DOD  leader  in  developing  practical  risk  assessment  and 
management  techniques  in  response  to  the  OSD  direction. 

The  process  of  risk  assessment  and  management  described  in  this  rjpo-*t 
will  provide  DARPA  with  the  ability  to  develop  a  justified  and  flexible  risk 
budget  unifying  and  balancing  the  requirements  of  its  critical  EEMIT  programs, 
minimizing  the  cost  overrun  potential  and  associated  program  failures. 

B.  THE  PROGRAM  MANAGEMENT  OFFICE  REQUIREMENT 

The  overall  management  of  risk  within  an  agency  can  be  approached  from 
two  perspectives,  either  "bottom-up"  or  "top-down.”  In  the  "bottom-up”  approach, 
risk  assessment  and  management  procedures  would  be  implemented  independently  by 
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each  program  manager,  based  on  his  unique  needs  and  perceptions.  Ultimately, 
the  risk  assessments  produced  in  this  manner  would  surface  as  part  of  each 
program's  budget  requirement,  hopefully  identified  and  justified  as  "risk 
budget."  Top-down  risk  management  would  originate  in  the  agency'3  senior 
management  level.  A  common  procedure  for  risk  assessment  and  management  would 
be  developed,  as  in  DARPA's  case,  by  the  Program  Management  Office,  and 
Implemented  in  all  programs  with  common  data  reporting  requirements  and 
milestones.  The  resultant  overall  assessment  of  risk  could  then  be  used  to 
implement  a  flexible  risk  budget.  Iterative  assessments  of  risk  in  each  pro¬ 
gram  could  then  be  used  to  continually  balance  requirements  between  programs 
in  accordance  with  agency-wide  priorities,  resulting  in  true  risk  management 
and  a  minimized  risk  budget . 

The  risk  assessment  and  management  process  described  in  this  report 
would  provide  the  Program  Management  Office  with  all  the  advantages  of  a  top- 
down  system,  while  the  individual  program  offices  retain  their  necessary 
element  of  internal  control  by  providing  the  initial  risk  assessments  at  the 
WBS-3  level  and  the  time-line  network,  as  well  as  the  opportunity  to  predict 
cost  and  schedule  changes  potentially  resulting  in  additional  resources.  In 
essence,  the  proposed  system  will  manage  only  risk  and  will  not  preempt  the 
program  managers'  functions. 

Additionally  this  process  will  provide  the  Program  Management  Office  with 
the  opportunity  to  participate  in  the  establishment  of  inter-program  priorities. 
With  this  insight  the  PMO  will  have  the  potential  to  identify  risk  causal 
factors  as  they  emerge,  as  well  as  risk  avoidance  and  reduction  alternatives. 

This  inter-program  overview  will  provide  the  PMO  with  the  capability  to  recommend 
risk  management  actions  to  the  Director  which  would  reduce  risk  and  minimise 
the  cost  overrun  potential  for  the  overall  EEMIT  program. 
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GROUP  I 

HIGH  RISK  PROGRAMS 
COST  FACTOR  RANGE  IN  0.5  TO  1.5 


POD 

’77 

'78 

’79 

DEF  Satellite  Comm.  System 

1.0 

.77 

.95 

HR  Maverick 

.98 

1.10 

1.17 

Laser  Maverick 

- 

.91 

- 

TR-1 

- 

1.0 

1.06 

DIVAD  Gun 

1.0 

1 .03 

1.04 

AALC  (Air  Cushioned  Vehicle) 

- 

1,003 

- 

F-18 

.99 

1.11 

1.87 

Tomahawk  (SLCM)  Missile 

.73 

1.06 

.64 

AN/TTC-39  Communications  Switch 

1.36 

1.37 

1.21 

Cannon  Launched  Guided  Projectile 

.89 

.98 

1,11 

Patriot  Missile  (S/iM-D) 

1,13 

1.19 

1.07 

YAH-64 

1.08 

1.10 

1.32 

CVN-68  Class  Aire ft  Carrier 

1.32 

1.36 

1.36 

LAMPS  MK  III 

1.08 

.90 

.92 

MK-48  Torpedo 

1.02 

.98 

.97 

PHM  Ship 

.54 

.54 

.55 

TACTAS 

.75 

.77 

1.64 

DOE 

Fast  Flux  Test  Facility 

- 

1.0 

1.0 

4.8  MW  Fuel  Cell  Demonstration  Unit 

1.0 

1.28 

1.28 

Fixed  Bed  Synthetic  Oil  Process  Development  Unit 

1.0 

1.004 

- 

Prassurized  Fluidized  Bed  Combined  Cycle  Plant 

1.0 

.97 

1.18 

14  MEV  Intense  Neutron  Source  Facility 

1.0 

1.0 

- 

Confinement  System  Poloidal  Diverter  Experiment 

- 

1.0 

- 

High  Energy  Laser  Facility 

2,41 

1.16 

1.16 

i 
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•77 

co 

•79 

DOE 

Mirror  Fusion  Test  Facility 

- 

1.0 

1.0 

Tokomak  Fusion  Test  Reactor 

1.11 

l.n 

1.33 

Intersecting  Storage  Accelerator,  400  GFV 
Electron  Volts 

1.0 

1.0 

Position  -  Electron  Joint  Project 

1.0 

1.0 

1.0 

High  Energy  Laser  Facility  (CA) 

1.25 

1.0 

1.0 

10  MW  Central  Receiver  Solar  Power  Plant 

1.0 

1.02 

1.0 

200  BEV  Accclereror 

.97 

.97 

- 

High  Btu  Synthetic  Pipeline  Gas  Demonstration 
Plant 

1.0 

l.U 

T  “  £ 

J.  •  4.W 

CF  DISTRIBUTION 


YEAR 

SAMPLE 

MEAN 

S  DEV 

S  DEV  TIMES  TWO 

77 

25 

1.06 

.33 

.66 

78 

32 

1.02 

.16 

.32 

79 

26 

1.11 

.27 

.54 

A- 3 


GROUP  II 

MEDIUM  RISK  PROGRAMS 
COST  FACTOR  RANGE  IN  0.0  TO  3.0 


DOD 

•77 

_!_78 _ 

’79 

CGN-38  Frigate 

1.57 

1.56 

1.55 

CH-53E  Helicopter 

1.61 

1.33 

1.38 

DD-963  Destoyer 

1.47 

1.61 

1.69 

CH-47  Helicoper  Modernization 

1.20 

1.13 

1.19 

LHA  Ship 

.96 

1.14 

1.21 

M-198  Howitzer 

1.80 

1.65 

1.47 

Trident  Support  Site,  Drydock  Phase  I  &  II 

1.0 

.86 

.92 

Trident  Support  Site,  Bangor.  WA,  Training 

1.11 

1.10 

1.17 

Facility,  Increment  1  &  2 

DOE 


High  Performance  Fuel  Lab 

1.25 

1.25 

— 

Combined  Cycle  Teat  Facility 

1.0 

3  .22 

- 

Component  Development  and  Integration  Facility 

1.19 

1.17 

1.17 

Component  Test  and  Integration  Unit 

1.0 

1.23 

1.4 

Donor  Solvent  Extraction  Pilot  Plant 

1.0 

1.0 

1.0 

Ebullated  BED  (H-Coal)  Pilot  Plant 

1.0 

1.27 

1.7 

Low  Btu  Fuel  Car  Demonstration  Plant  -  Small 
Industrial 

- 

1.0 

- 

Fusion  Materials  Irradiation  Test  Facility 

- 

1.0 

1.27 

S8G  Prototype  Propulsion  Plant 

1.23 

1.23 

- 

0TEC  1  Ocean  Test  Facility 

- 

1.15 

.86 

Additional  Facilities,  High  Level  Waste  Handling 

4  Storage 

1.05 

1.05 

- 

Additional  Facilities,  High  Level  Waste  Storage 

1.0 

1.00 

1.0 

Additional  High  Level  Waste  Storage  Facilities 

1.0 

1.0 

1.0 

A-4 


’77 

•78 

•79 

DOE 

Additional  Waste  Concentration  and  Salt  Lake 
Storage  Facilities 

.92 

.92 

.92 

Fluorine  Dissolution  Process  and  Fuel  Receiving 
Improvements 

3.31 

1.0 

1.29 

High  Level  Waste  Storage  and  Handling  Facilities 

1.0 

1.0 

1.0 

High  Level  Waste  Storage  Facilities 

- 

1.0 

1.0 

High  Level  Waste  Storage  Tanks  and  Waste 

Management  Facilities 

1.0 

1.0 

1.0 

Improvements  to  Waste  Management  and  Materials 
Processing  Facilities 

- 

1.0 

- 

New  Waste  Calcining  Facility 

3.25 

1.25 

1.25 

Additional  Facility  for  Enriched  Uranium 

Production 

1.5 

.98 

1.06 

Cascade  Upratlng  Program,  Gaseous  Diffusion 

Plants 

1.96 

.92 

.92 

Centrifuge  Facility  Modification 

- 

1.0 

1.0 

Centrifuge  Plant  Demonstration  Facility 

- 

1.0 

1.0 

DP  Site  PU  Processing  Facility 

2.5 

1.0 

1.0 

New  Plutonium  Recovery  Facility 

2.97 

1.44 

1.54 

Weapons  Safeguards 

- 

1.0 

1.0 

CF  DISTRIBUTION 

YEAR 

SAMPLE 

MEAN 

S  DEV 

S  DEV  TIMES  TWO 

77 

27 

1.48 

.72 

1.44 

78 

32 

1.13 

.20 

.40 

79 

29 

1.14 

.32 

.64 

GROUP  III 


NASA  PROGRAMS  1980 
COST  FACTOR  RANGE  IN  1.2  TO  2.0 


'80 

Galileo  1.98 
International  Solar  Polar  Mission  1.51 
Landsat  D  1.55 
Space  Telescope  1^43 
Space  Transportation  System  1.29 


CF  Distribution 


Sample 

5 

Mean 

1.55 

Standard  "Deviation 

.26 

S  Dev  times.!  two 

.52 

A- 6 


GROUP  IV 

HIGH  RISK  PROGRAMS 
COST  FACTOR  RANGE  IN  1.5  TO  3.0 


*77 

'78 

•79 

A-10  Aircraft 

1.82 

1.88 

1.93 

Advanced  Airborne  Command  Post 

1.48 

1.55 

.98 

Airborne  Warning  and  Control  System 

1.32 

1.56 

1.56 

EF-111A 

1.16 

1.68 

1.84 

F~15 

1.77 

1.79 

1.81 

F-16 

2.28 

2.48 

2.49 

Improved  Hawk  Missile 

2.09 

2.02 

2.01 

Stinger  Missile 

1.99 

2.78 

2.62 

UH-60A 

1.50 

1.57 

1.59 

XM-1  Tank 

1.01 

2.17 

2.29 

F-14 

1.73 

1.96 

1.98 

PHALANX  CIWS 

1.88 

1.97 

2.29 

SSN-688 

1.84 

1.67 

1.88 

Surtass  Array  Sensor 

1.86 

1.89 

2.02 

Trident  Sub/Mlsslle  System 

1.79 

1.93 

2.05 

Roland  Missile 

1.66 

1.90 

2.14 

Sparrow  AIM-7F  (Navy) 

2.11 

2.28 

3.26 

CF  DISTRIBUTION 


YEAR 

SAMPLE 

MEAN 

S  DEV 

S  DEV  TIMES  TWO 

77 

17 

1.72 

.34 

• 

0* 

00 

78 

17 

1.95 

.33 

.66 

A- 7 


79 


17 


2.04 


49 


98 


1 


DOD  DEVELOPMENTAL 

ADJUSTED  HIGH  RISK  PROGRAM  COST  FACTOk  DISTRIBUTIONS 


•80 

'79 

•78 

’77 

A- 10  Aire  -aft 

2.27 

1.92 

1.87 

1.81 

Advanced  Airborne 

Command  Past 

2.67 

1.63 

1.68 

1.60 

Airborne  Warning  and 
Control  System 

1.62 

1.54 

1.54 

1.52 

DEF  Satellite  Communi¬ 
cation  System 

1.20 

0.95 

0.77 

1.00 

EF-111A 

2.14 

2.55 

1.59 

1.14 

F-15 

1.76 

1.70 

1.69 

1.67 

F-16 

2.20 

1.59 

1.60 

1.40 

HR  Maverick 

1.50 

1.13 

1.09 

.99 

Laser  Maverick 

1.16 

1.16 

AIM-9L  Sidewinder 

Air  Force 

Navy 

3.52 

3.90 

2.93 

2.96 

2.73 

2.83 

2.62 

2.50 

AIM-7F  Sparrow 

Air  Force 

Navy 

2.67 

2.82 

2.40 

2.66 

2.26 

2.54 

2.16 

2.37 

AH- 6  4 

1.56 

1.32 

1.10 

1.08 

AN/TTC-39 

1.54 

1.34 

1.31 

1.30 

UH-60A 

2.60 

1.59 

1.37 

1.50 

Cannon  Launched 

Guided  Projectile 

1.45 

1.21 

1.08 

.99 

DIVAD 

1.27 

1.02 

1.01 

Patriot  (SAM-D) 

1.86 

1.42 

1.39 

1.33 

Roland 

2.00 

2.03 

1.80 

i.56 

Stinger 

2.13 

2.28 

2.39 

1.63 

XM-1  Tank 

1.74 

1.29 

1.17 

1.01 

CAPTOR 

3.40 

4.63 

3.81 

3.12 

« 


DARPA 


HIGH  TECHNOLOGY-HIGH  RISK  PROGRAMS 


'78 

'79 

'80 

'81 

Ceramic  Turbine 

1.13 

1.13 

1.00 

1.00 

— 

— 

— 

1.00 

Teal  Ruby 

1.58 

1.00 

1.00 

1.00 

2.51 

2.87 

1.25 

1.00 

X-Wing 

— 

— 

1.00 

1.00 

— 

— 

1.36 

— 

HIMAG 

— 

1 .57 

1.20 

1.20 

— 

1.19 

- — 

— 

SIAM 

— 

1.00 

1.00 

1.00 

— 

— 

2.10 

2.10 

ACCAT 

1.00 

1.00 

1.00 

1.00 

— 

2.35 

1.08 

1.00 

— 

— 

1.25 

1.00 

Mini-Halo 

— 

— 

1.00 

1,00 

Talon  Gold 

— 

— 

1.13 

1.00 

ALPHA 

— 

— 

1.01 

1.00 

Forward  Swept  Wing 

— 

— 

— 

1.13 

Air  Warfare 

— 

1.00 

1.00 

1.00 

— 

— 

1.00 

1.00 

Anti-Armor 

— 

1.37 

1.68 

1.00 

— 

1.16 

1.42 

— 

— — 

— 

1.80 

— 

Teal  Rain 

— 

— 

— 

1.02 

— 

— 

— 

1.24 

Assault  Breaker 

— 

— 

1.01 

1.00 

— 

— 

1.10 

1.37 

— 

— 

1.46 

1.53 

A-IO 


